Staged Training Report ✓ Complete

Run ID: shoulder_pan_with_lr_sweep_on_plateau
Generated: 2026-02-06 13:57:08
Stages Completed: 10
Total Elapsed Time: 17:33:56

Timing Summary

Stage Plateau Sweeps Sweep Time Training Time Stage Total
Stage 1 1 00:06:49 00:02:19 00:09:08
Stage 2 1 00:09:55 00:02:20 00:12:16
Stage 3 6 01:39:42 01:04:15 02:43:57
Stage 4 3 00:51:18 01:05:00 01:56:18
Stage 5 0 00:00:00 00:01:30 00:01:30
Stage 6 5 01:21:28 00:11:00 01:32:28
Stage 7 5 01:24:32 01:08:11 02:32:44
Stage 8 5 01:25:29 01:05:57 02:31:26
Stage 9 4 01:08:38 01:06:42 02:15:21
Stage 10 7 01:58:59 01:11:39 03:10:39
TOTAL 37 10:06:54 06:58:56 17:05:51

Plateau Sweep Details

Total Sweeps: 37
Stages with Sweeps: 9 of 10
Total Sweep Time: 10:06:54
Average Sweep Duration: 00:16:24

Stage 1: 1 sweep

LR Progression: 1.0e-04 → 1.0e-03

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 2,500 00:01:35 1.00e-03 00:06:49

Stage 2: 1 sweep

LR Progression: 1.0e-04 → 1.0e-03

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 7,500 00:01:36 1.00e-03 00:09:55

Stage 3: 6 sweeps

LR Progression: 1.0e-04 → 1.0e-03 → 1.0e-03 → 1.0e-03 → 1.0e-03 → 1.0e-03 → 1.0e-03

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 10,500 00:01:39 1.00e-03 00:16:42
2 13,000 00:19:59 1.00e-03 00:16:30
3 15,500 00:38:13 1.00e-03 00:16:30
4 18,000 00:56:23 1.00e-03 00:16:26
5 20,500 01:14:26 1.00e-03 00:16:59
6 23,000 01:33:02 1.00e-03 00:16:33

Stage 4: 3 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 119,000 00:01:41 1.00e-06 00:17:14
2 121,500 00:20:34 1.00e-06 00:17:01
3 124,000 00:39:09 1.00e-06 00:17:02

Stage 5: 0 sweeps

LR Progression: 1.0e-04 (unchanged)

Stage 6: 5 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-05 → 1.0e-05

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 160,500 00:01:37 1.00e-06 00:17:16
2 163,000 00:20:30 1.00e-06 00:16:30
3 165,500 00:38:10 1.00e-06 00:16:30
4 168,000 00:55:50 1.00e-05 00:16:17
5 170,500 01:13:17 1.00e-05 00:14:53

Stage 7: 5 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 181,500 00:01:38 1.00e-06 00:17:16
2 184,000 00:20:30 1.00e-06 00:17:07
3 186,500 00:39:12 1.00e-06 00:17:06
4 189,000 00:57:57 1.00e-06 00:16:33
5 191,500 01:16:11 1.00e-06 00:16:28

Stage 8: 5 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-05

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 266,750 00:01:37 1.00e-06 00:17:17
2 269,250 00:20:30 1.00e-06 00:17:34
3 271,750 00:39:38 1.00e-06 00:17:14
4 274,250 00:58:26 1.00e-06 00:16:53
5 276,750 01:16:59 1.00e-05 00:16:29

Stage 9: 4 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 370,500 00:01:38 1.00e-06 00:17:26
2 373,000 00:20:41 1.00e-06 00:17:19
3 375,500 00:39:39 1.00e-06 00:16:50
4 378,000 00:58:10 1.00e-06 00:17:01

Stage 10: 7 sweeps

LR Progression: 1.0e-04 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-06 → 1.0e-04

Sweep # Triggered At (samples) Wall Time Selected LR Duration
1 445,250 00:01:38 1.00e-06 00:17:41
2 447,750 00:20:56 1.00e-06 00:17:33
3 450,250 00:40:08 1.00e-06 00:17:00
4 452,750 00:58:50 1.00e-06 00:16:55
5 455,250 01:17:25 1.00e-06 00:17:01
6 457,750 01:36:05 1.00e-06 00:16:34
7 460,250 01:54:22 1.00e-04 00:16:11

Stage Results

Stage Best Loss Stop Reason Samples Trained Time Sweeps LR (Initial→Final)
Stage 1 0.091311 divergence 2,500 00:09:08 1 1.0e-04→1.0e-03
Stage 2 0.091220 divergence 2,500 00:12:16 1 1.0e-04→1.0e-03
Stage 3 0.037263 divergence 98,750 02:43:57 6 1.0e-04→1.0e-03
Stage 4 0.037085 divergence 119,250 01:56:18 3 1.0e-04→1.0e-06
Stage 5 0.048573 divergence 2,250 00:01:30 0 1.0e-04
Stage 6 0.051523 divergence 15,000 01:32:28 5 1.0e-04→1.0e-05
Stage 7 0.053736 time_budget (60.0 min limit) 113,250 02:32:44 5 1.0e-04→1.0e-06
Stage 8 0.043251 divergence 99,000 02:31:26 5 1.0e-04→1.0e-05
Stage 9 0.043426 time_budget (60.0 min limit) 79,250 02:15:21 4 1.0e-04→1.0e-06
Stage 10 0.041176 time_budget (60.0 min limit) 154,750 03:10:39 7 1.0e-04

Total Plateau Sweeps: 37

Stop Reason Breakdown

Best Checkpoint

Name: best_model_auto_session_so101_should_pan_500_stage10_train_304_00600750_cont_val_0.041176.pth
Stage: 10
Hybrid Loss (full session): 0.056461

Learning Rate Timeline with Plateau Sweeps

Stage Progression

Stage Orig Loss Train Loss Time Samples Stop Reason
1 0.104286 0.091311 00:09:08 2500 divergence
2 0.104652 0.091220 00:12:16 2500 divergence
3 0.077820 0.037263 02:43:57 98750 divergence
4 0.073566 0.037085 01:56:18 119250 divergence
5 0.075535 0.048573 00:01:30 2250 divergence
6 0.062316 0.051523 01:32:28 15000 divergence
7 0.061903 0.053736 02:32:44 113250 time_budget (60.0 min limit)
8 0.060444 0.043251 02:31:26 99000 divergence
9 0.059957 0.043426 02:15:21 79250 time_budget (60.0 min limit)
10 ⭐ 0.056461 0.041176 03:10:39 154750 time_budget (60.0 min limit)

Hybrid Loss Over Original Session (per Stage)

Stage 1 - Hybrid Loss: 0.104286

Stage 2 - Hybrid Loss: 0.104652

Stage 3 - Hybrid Loss: 0.077820

Stage 4 - Hybrid Loss: 0.073566

Stage 5 - Hybrid Loss: 0.075535

Stage 6 - Hybrid Loss: 0.062316

Stage 7 - Hybrid Loss: 0.061903

Stage 8 - Hybrid Loss: 0.060444

Stage 9 - Hybrid Loss: 0.059957

Stage 10 (Best) - Hybrid Loss: 0.056461

Sample Counts

Cumulative Across All Stages

Per Stage

Stage 1 - Total Samples: 2,500

Stage 2 - Total Samples: 2,500

Stage 3 - Total Samples: 98,750

Stage 4 - Total Samples: 119,250

Stage 5 - Total Samples: 2,250

Stage 6 - Total Samples: 15,000

Stage 7 - Total Samples: 113,250

Stage 8 - Total Samples: 99,000

Stage 9 - Total Samples: 79,250

Stage 10 (Best) - Total Samples: 154,750

Best Checkpoint Inference

Selected Frame 3

Action 0

Action 1

Action 2

Random Observations

Observation 224

Action 0
Action 1
Action 2

Observation 106

Action 0
Action 1
Action 2